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FOREWORD 


Each  of  the  military  services  has  made  a  commitment  to  developing  and 
implementing  automated  information  and  decision  support  systems  (DSS) .  How¬ 
ever,  successful  transfer  of  this  technology  is  ultimately  contingent  on 
users'  acceptance  and  use  of  the  technology. 

This  report  presents  a  comprehensive  overview  of  the  problems  in  user 
acceptance  from  the  viewpoints  of  both  researchers  and  military  personnel  who 
have  experience  in  aid  design  and  implementation.  The  report  should  be  useful 
to  military  DSS  builders  and  developers  in  helping  them  to  successfully  trans¬ 
fer  automation  technology  to  the  user. 

The  initiative  for  the  Workshop  on  User  Acceptance  came  from  the  Joint 
Services  Working  Group  on  Decision  Aiding  (JSWGDA) ,  a  subgroup  of  the  Decision 
Aids  subpanel  of  the  Joint  Directors  of  Laboratories  Technology  Panel  for  . 

A  major  objective  of  the  Working  Group  is  to  promote  the  exchange  of  informa¬ 
tion  underlying  advances  in  C^  decision  aiding  and  facilitate  joint  service 
research  and  activities  in  this  area.  The  workshop  is  such  a  joint  activity 
with  representatives  from  the  services  participating. 

The  project  was  conducted  under  research  task  1.4.4,  Evaluating  and 
Enhancing  Command  Staff  Operations.  It  was  cosponsored  by  the  U.S.  Army 
Research  Institute,  JSWGDA,  and  the  Combined  Arms  Combat  Developments  Activ¬ 
ity.  The  results  of  this  report  were  briefed  on  July  7,  1987,  to  the  Joint 
Services  Working  Group  on  Decision  Aiding,  various  government  contractors,  and 
other  personnel  from  the  tri-services.  The  report  will  be  used  by  the  JSWGDA 
to  identify  areas  for  joint  services  research. 


EDGAR  M.  JOHNSON 
Technical  Director 
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USER  ACCEPTANCE  AND  FIELD  IMPLEMENTATION  OF  DECISION  SUPPORT  SYSTEMS 


I 

EXECUTIVE  SUMMARY 


Requirement : 

To  identify  the  problems  relating  to  the  acceptance  of  military  decision 
support  systems  (DSS)  and  to  make  recommendations  for  addressing  each  of  these 
problems;  to  identify  problems  relating  to  involving  the  user  in  design  and 
implementation  and  to  make  recommendations  for  addressing  each  of  them;  and 
to  discuss  mechanisms  for  rield  implementation  that  would  enhance  user 
acceptance . 


Procedure : 

The  U.S.  Army  Research  Institute,  the  Joint  Services  Working  Group  on 
Decision  Aiding,  and  the  U.S.  Army  Combined  Arms  Combat  Developments  Activity 
sponsored  an  invitational  workshop  to  explore  factors  that  affect  user  accep¬ 
tance.  Knowledgeable  participants,  with  experience  in  the  design,  evaluation, 
and  implementation  of  military  DSS,  as  well  as  researchers  in  these  areas, 
were  identified  and  invited  to  the  2 -day  workshop.  The  14  participants  in¬ 
cluded  representatives  from  the  Army,  Air  Force,  and  government  contractor 
community. 

Participants  were  divided  into  two  groups,  each  representing  a  mix  of 
military,  government  employees,  and  government  contractors.  Each  group  in¬ 
dependently  discussed  assigned  topics  relating  to  the  workshop  objectives. 

Summaries  of  each  subgroup's  discussions  were  presented  to  the  larger 
group  and  discussed  in  this  forum.  This  report  presents  the  material  devel¬ 
oped  by  this  method. 


Findings : 

a.  Twenty-two  causes  of  problems  of  user  acceptance  were  identified,  and 
recommendations  for  addressing  each  were  developed.  Problems  were  categorized 
into  those  involving  perceived  lack  of  utility,  difficulty  using  the  system, 
and  damage  to  the  user. 

b.  General  recommendations  for  addressing  these  problems  included 

(1)  early  and  ongoing  involvement  of  the  users  in  the  development  of  require¬ 
ments,  in  system  design,  and  ir.  development  and  implementation;  (2)  common 
user  interface  across  aids  and  systems;  (3)  evolutionary  design  and  develop¬ 
ment  of  aids;  (4)  education  to  alter  erroneous  perceptions  and  adequate  train¬ 
ing  in  the  vise  of  the  system:  {c''*  development  of  formal  organizational  links 


vii 


I 


between  users,  combat  developers,  and  aid  builders;  and  (6)  use  of  adaptive 
design  and  rapid  prototyping  if  they  can  be  integrated  into  the  materiel 
acquisition  process. 

c.  Most  of  the  recommendations  in  the  report  are  best  accomplished 
through  careful  organizational  management  of  the  design  and  implementation  of 
the  system. 

Utilization  of  Findings: 

The  findings  will  be  useful  to  the  combat  developers  and  builders  tasked 
with  designing,  implementing,  and  fielding  military  decision  support  systems. 
The  findings  indicate  that  organizational  mechanisms  are  needed  to  address 
many  of  the  user  acceptance  problems  that  were  identified. 
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INTRODUCTION 


The  military  services  have  made  a  commitment  to  the  development  and  use  of 
automated  information  and  decision  support  systems.  Given  the  enormous  invest¬ 
ment  of  resources  in  such  technology,  it  is  important  to  ensure  its  efficient 
transfer  into  the  operational  environment.  However,  there  are  documented  cases 
of  military  technology  innovations  which  have  not  achieved  user  acceptance  and 
which  have  been  subsequently  misused  and  even  rejected  by  users.  It  would  be 
useful  then,  to  be  able  to  predict  and  control  the  factors  that  adversely 
affect  user  acceptance  of  such  technologies.  This  report  documents  the  results 
of  a  workshop  that  explored  factors  contributing  to  user  non-acceptance  and 
generated  recommendations  for  controlling  these  factors. 

The  US  Army  Research  Institute  (ARI),  the  Combined  Arms  Conbat  Developments 
Activity  (CACDA),  and  the  Joint  Services  Working  Group  on  Decision  Aiding  spon¬ 
sored  a  Workshop  to  discuss  factors  that  affect  user  acceptance.  The  objec¬ 
tives  of  the  Workshop  were  (1)  to  explore  the  factors  that  affect  user 
acceptance  of  military  decision  aids/support  systems;  (2)  to  develop  solutions 
to  user  acceptance  problems;  and  (3)  to  discuss  mechanisms  for  field  implemen¬ 
tation  that  may  enhance  user  acceptance.  In  order  to  accomplish  these  objec¬ 
tives,  knowledgeable  participants,  who  had  experience  with  aid  development  and 
implementation  or  who  had  done  research  in  user  acceptance,  were  invited  to  the 
Workshop.  This  report  summarizes  the  ideas  and  recommendations  of  the  Workshop 
participants . 

A  number  of  definitions  of  the  term  "decision  support  systems"  (DSS)  have 
been  advanced.  For  the  Workshop,  the  term  was  defined  as  computer  software 
that  supports  decision  making.  "Support"  was  defined  very  generally  so  that 
any  aid  or  system  that  participants  wanted  to  discuss  would  be  included.  As 
it  is  used  here,  "decision  support  systems"  is  a  very  broad  term.  It  would 
inc'  -de  one  simple  program  designed  to  aid  one  staff  user  in  accomplishing  one 
task  as  well  as  a  complex  multi-purpose  integrated  system  which  supports  multi¬ 
ple  users  working  multiple  tasks,  sharing  information  within  or  between  battle¬ 
field  functional  areas.  Some  of  the  user  acceptance  issues  will  be  the  same 
regardless  of  the  complexity  or  size  of  the  aid,  while  other  issues  will  be 
specific  to  certain  types  of  aids.  A  goal  of  the  workshop  was  to  identify  a 
comprehensive  set  of  user  acceptance  issues,  and  defining  DSS  very  broadly 
insures  that  the  issues  identified  will  be  as  comprehensive  as  possible. 
Throughout  the  report,  as  in  the  Workshop,  "aid"  and  "DSS"  were  used  inter¬ 
changeably  even  though  "DSS"  is  the  more  comprehensive  term. 

The  material  in  the  report  is  slanted  toward  the  Army  because  of  the  make¬ 
up  of  the  Workshop  group.  However,  the  recommendations  are  generally  applica¬ 
ble  because  similar  user  acceptance  problems  exist  in  all  three  services. 


APPROACH 


The  workshop  brought  together  14  participants  from  ARI,  CACDA,  TFLADOC^ 
Analysis  Command  (TRAC),  the  Air  Force  Institute  of  Technology,  US  Army  Signal 
Center  and  School,  and  several  government  contractors.  (See  Appendix  A  for  a 
list  of  participants).  Participants  were  divided  into  two  groups  each  repre¬ 
senting  a  mix  of  military,  government  employees,  and  government  contractors. 
These  groups  were  small  enough  to  permit  all  to  participate  and  the  mix  repre¬ 
sented  a  diversity  of  perspectives,  expertise,  and  experience. 

Each  group  independently  discussed  the  following  topics: 

1.  Definition  of  the  User. 

2.  Specific  factors  that  affect  user  acceptance. 

3.  Strategies  for  addressing  each  of  these  factors. 

4.  Problems  with  involving  users  in  tne  design  and  evaluation 
of  automated  systems. 

5.  Approaches  for  addressing  user  involvement  problems. 

Summaries  of  each  subgroups'  discussions  were  presented  to  the  whole  group  and 
discussed  again.  The  following  report  presents  the  material  developed  through 
this  method.  The  material  does  not  necessarily  represent  a  consensus  opinion, 
but  represents  majority  or  significant  minority  opinions. 
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DEFINING  THE  USER 


Before  we  could  discuss  user  acceptance,  we  needed  to  clarify  who  we  mean 
by  "the  user".  There  can  be  many  different  types  of  users  of  the  same  aid,  and 
these  users  can  vary  in  a  number  of  ways. 

•  There  are  different  levels  of  users  for  different  aspects  of 
the  aids: 

•  End  user  or  aid  operator 

•  Decision  maker:  commander  and  staff 

•  Organizations 

In  some  instances  these  users  can  be  the  same  person.  For  example, 
the  G3  (i.e.,  the  operations  officer)  could  both  interact  physi¬ 
cally  with  the  aid  and  use  the  aid's  output  in  his  recommendations. 

•  Within  each  of  th^sp  levels,  individual  users  can  differ.  For  exam¬ 
ple,  they  can  be  experienced  or  inexperienced,  commander  or  staff, 

G 2  (i.e.,  the  intelligence  officer)  or  G3.  Table  1  presents  vari¬ 
ables  on  which  users  differ  and  which  may  influence  the  users'  per¬ 
formance  or  acceptance  of  an  aid. 

•  There  can  be  "within-user"  differences.  That  is,  the  same  user 
can  differ  on  a  variable  depending  on  the  task,  environment,  or 
with  the  passage  of  time.  For  example,  the  experience  level  of  the 
user  will  change  the  more  he  uses  the  system. 

•  Use  of  an  aid  may  change  the  task  structure  or  work  flow  so  that  the 
intended  user  may  not  end  up  being  the  actual  user.  For  example, 

it  may  have  teen  originally  intended  that  a  staff  officer  would 
physically  interact  with  the  aid  upon  instructions  from  a  commander. 
However,  in  actual  use  the  commander  himself  may  interact  with  the 
aid  to  get  the  information  he  needs. 

•  Future  aids  will  be  part  of  larger  integrated  systems  with  multiple 
functions,  users  and  environments.  For  example,  Brigade  Planner 
(Diaz  &  Smith,  1936),  developed  by  TRAC,  supports  multiple  tasks  for 
different  users.  In  the  AirLand  Battle  Concept,  synchronization  is 
an  important  requirement  for  battle  success.  To  accomplish  this 
synchronization,  automated  systems  of  the  future  may  have  a  common 
user  interface  to  facilitate  the  sharing  of  information  and  battle 
management  plans.  The  same  system  interface  may  be  used  by  users 
from  different  functional  areas  and  even  different  services. 
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Table  1 


Individual  Differences  That  May  Affect  Decision  Making  Performance 


Demographic  Characteristics 
Age 

Gender 

Rank/Command  Level 
Education 

Personality  Characteristics 
Decision  Making  Style 
Cognitive  Style 
Learning  Style 
Risk  Taking  Propensity 
Motivation 
Locus  of  Control 

Skills/ Abilities 
|  Task  Expertise 

Knowledge  of  Task  Requirements 
Skill/Experience  with  the  System 
Training 

Spatial  and  Verbal  AhMity 
Intelligence 

Preferences 

Goals 

Preferences  for  Display  Format 
Preferred  Sensory  Modality 
Preferred  Communication  Mode 


The  same  system  can  have  many  different  types  users.  Acceptance  by  one  user 
does  not  mean  acceptance  by  all  users.  Different  users  have  different  types  of 
user  acceptance  problems  and  different  users  are  needed  to  evaluate  different 
aspects  of  user  acceptance. 

The  question  "Who  is  the  user?"  has  implications  for  other  aspects  of  sys¬ 
tem  development.  Most  aid  decision  and  evaluation  handbooks  recommend  that  the 
intended  users  be  employed  to  develop  aid  design  requirements  and  be  used  to 
evaluate  the  aid.  However,  Workshop  participants  felt  that  the  diversity  of 
potential  users  is  a  major  problem  in  the  development  of  automated  support 
systems.  That  is,  which  user  should  be  selected  for  design  and  evaluation 
input?  The  user  one  selects  will  affect  aid  design  and  evaluation  results.  If 
different  users  produce  markedly  different  design  requirements  and  evaluation 
results,  then  this  suggests  that  results  obtained  with  one  or  two  users  are  not 
generalizable  to  the  whole  population  of  potential  users.  This  is  a  problem 
which  is  not  being  addressed  sufficiently  and  often  not  even  recognized  as  a 
problem.  Designers  and  evaluators  do  not  specify  what  characteristics  the  user 
must  have  who  is  to  assist  them  and  developers  don't  spend  time  selecting  the 
proper  user  to  send  to  the  designer  or  evaluator. 
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FACTORS  THAT  AFFECT  USER  ACCEPTANCE  AND  RECOMMENDATIONS 
FOR  ADDRESSING  THESE  FACTORS 


Workshop  participants  identified  22  factors  that  can  adversely  affect  the 
user  acceptance  of  an  aid  and  they  made  recommendations  for  dealing  with  each 
factor.  Table  2  presents  a  summary  of  the  factors  and  recommendations.  Not 
all  of  the  participants  agreed  with  each. 

Perceived  Lack  of  Utility 

Workshop  participants  thought  that  perceived  utility  was  the  most  important 
factor  in  acceptance  of  the  aid.  If  the  system  clearly  supports  improved  deci¬ 
sion  making  and  meets  user  needs  then  other  factors  adverse  to  acceptance  may 
be  overridden.  Perceived  lack  of  utility  has  two  major  causes.  The  aid  may 
indeed  lack  utility.  This  is  discussed  in  items  one  through  three  below.  On 
the  other  hand,  if  the  aid  does  meet  user  needs,  then  the  lack  of  user  accep¬ 
tance  stems  from  erroneous  perceptions  of  the  aid.  This  is  discussed  in  items 
four  through  thirteen. 

1 .  The  aid  lacks  utility. 

Lack  of  utility  in  an  aid  can  be  a  problem  of  requirements  definition,  and 
requirements  definition  is  not  an  easy  task.  Users  may  not  know  what  they 
need,  or  how  to  specify  their  needs;  users  may  not  agree  on  their  needs;  they 
are  not  good  at  predicting  future  needs;  needs  may  change  with  the  incorpora¬ 
tion  of  the  aid;  or  different  users  of  the  aid  tnay  have  different  needs. 

It  is  not  unusual  for  aid  development  to  be  driven  by  technology  rather 
than  requirements.  Builders  learn  about  a  promising  technology  and  look  for 
potential  applications  of  the  technology.  The  aid  is  selected  for  development 
not  because  it  fills  a  user  need,  but  because  it  is  an  application  of  a  tech¬ 
nology  the  builder  wants  to  use. 

Recommendations:  Clearly,  there  is  a  need  for  user  involvement  in  require¬ 
ments  analysis.  One  approach  to  this  problem  is  through  adaptive  design  -  where 
the  Subject  Matter  Expert  (SHE)  is  the  designer  and  does  his  own  requirements 
specification.  Another  technique  is  rapid  prototyping.  Here  a  prototype  of 
the  aid  is  developed,  tested  by  the  users,  rapidly  modified,  and  tested  again, 
etc.  The  rationale  is  that  it  is  easier  for  a  user  to  react  to  a  concrete  aid 
than  to  imagine  what  he  needs.  It  is  an  "I  know  what  I  like  when  I  see  it" 
approach.  One  general  problem  with  user  involvement  in  rapid  prototyping  is 
that  users  are  needed  on  an  ongoing  and  long  term  basis.  Another  problem  is 
related  to  the  generalizability  of  design  and  evaluation  results  obtained  from 
users.  Typically,  only  one  or  two  users  are  used  for  rapid  prototyping,  and 
unless  they  are  chosen  carefully,  it  will  not  be  clear  to  which  user  population 
the  results  can  be  generalized. 


Table  2 

Summary  of  User  Acceptance  Factors  and  Recommendations 


General  Factor 

Cause 

Recommendations 

Perceived 
lack  of 
utility 

1. 

Aid  lacks  utility 

Involve  user  in  design 
Adaptive  design 

Rapid  prototyping 

Test  &  evaluation 

2. 

Aid  is  not  always  appropriate 

Education  about  aid's 
limitations 

Explanation  capability 

3. 

Aid  requires  organizational 
restructuring 

Task  &  information  flow 
analysis 

4. 

Incompatibility  of  aid's  and 
user's  problem  representation 

Make  compatible 
Explanation  capability 
Involve  users  in  design 

5. 

Unfamiliar  decision  procedures 

Involve  users  in  design 

6. 

Perception  of  builders 

Demonstration  of  utility 

7. 

Premature  demonstration  of  aid 

Break-in  period  precedes 
demonstration 

8. 

Aid  does  not  speak  "green  suit" 

Involve  users  in  design 

9. 

Lack  of  confidence 

Explanation  capability 
Demonstration  of  utility 

10. 

"Aid  will  go  away" 

System  integration  of  aid 
Senior  level  commitment 

11. 

Inadequate  training 

Pretesting  to  determine 
training  needs 

12. 

Personnel  change 

On-going  training 

Embedded  training 

Standard  interface 

13. 

Bad  experience  with  salesmen 

Utility  demonstration 

Test  and  evaluation 

Damage 

14. 

Distrust  of  new  technology 

Training 

to  user 

15. 

Loss  of  control  over  decisions 

Explanation  capability 
Training  on  aid's 

1 imitations 


16. 

Damage 

to 

skills 

and 

expertise 

Override  capability 

Maintain  a  backup  system 

17. 

Damage 

to 

career 

or 

status 

Organization- sanctioned 

18. 

Increased 

workload 

aids 

Workload  analysis  and  task 

19. 

"Real 

men 

do  not 

use 

keyboards" 

allocation  so  aids  do  not 
make  more  work 
Demonstration  of  utility 

Top  down  implementation 
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Table  2  (Continued) 


Summary  of  User  Acceptance  Factors  and  Recommendations 


General 

Factor 

Cause 

Recommendations 

Aid  is 

20. 

Bad  interface  design 

User  involvement  in  design 

hard  to 

21. 

Hardware  &  software 

Standard  user  interface 

use 

22. 

incompatibility 

Aid  packages  are  not  integrated 

with  "device  drivers"  to 
handle  incompatibilities 
Organizational  mechanism 

common  functional  standards 


Extensive  testing  and  evaluation  of  the  aid  or  system  in  the  conceptual, 
design,  prototype,  and  fielding  stages  will  also  ensure  that  the  aid  is  a  use¬ 
ful  and  reliable  tool.  Results  of  the  testing  can  then  be  used  to  demonstrate 
to  users  that  the  aid  can  help  them  and  to  increase  their  confidence  in  the 
aid. 


2.  The  aid  is  not  always  appropriate. 

Even  if  the  aid  does  provide  the  type  of  assistance  needed,  the  user  may 
not  be  sure  that  the  quality  of  the  aid's  output  will  always  match  what  could 
have  been  done  unaided.  The  user  may  feel  that  the  aid  will  not  be  correct  or 
applicable  in  all  situations,  but  that  he  does  not  know  enough  about  the  aid  to 
recognize  the  errors  or  situations  in  which  the  aid  results  should  not  be  used. 
A  related  problem  is  a  perception  that  novice  users  will  be  too  accepting  of 
the  aid,  not  be  able  to  evaluate  the  aid's  products,  and  therefore  use  it 
uncritically . 

Recommendations.  In  the  development  and  testing  of  the  aid,  special  atten¬ 
tion  should  be  paid  to  the  limitations  and  boundaries  of  the  aid  and  mechanisms 
for  identifying  errors.  The  limitations  of  the  system  and  identification  of 
errors  should  be  treated  explicitly  in  the  training  course  and  in  training 
embedded  in  the  aid.  By  incorporating  an  explanation  capability  in  the  aid, 
novice  users  will  become  more  knowledgeable  and  not  indiscriminately  accepting 
of  the  aid's  results.  Algorithms  should  be  well  documented. 

3 .  The  aid  requires  organizational  restructuring. 

Use  of  the  aid  may  result  in  a  change  in  information  flow  and  task  struc¬ 
ture.  If  these  necessitate  a  major  organizational  change  the  aid  may  not  be 
used.  On  the  other  hand,  at  least  some  organizational  changes  will  be  neces¬ 
sary,  and  acceptance  will  depend  on  the  individual  organization's  flexibility 
in  coping  with  change. 

Recommendations.  Builders  should  attend  to  existing  task  and  information 
flow  structure  and  design  aids  that  fit  in.  As  discussed  previously  automation 
may  change  the  way  decisions  are  made,  and  at  some  point  a  restructuring  of  the 
present  military  organization  may  be  in  order.  However,  since  it  is  not  clear 
how  automation  will  change  the  existing  organization,  such  changes  will  not 
occur  in  the  near  future.  Incompatibility  with  organizational  structure  will 
most  likely  result  in  scraping  the  aid. 

4 .  Incompatibility  between  the  aid's  representation  of  the  problem  and 
the  user's  representation. 

The  system’s  conceptualization  and  representation  of  the  decision  problem 
may  not  look  like  that  with  which  the  user  is  familiar.  It  may  not  seem  natu¬ 
ral  or  to  fit  the  problem,  or  it  may  not  follow  the  standard  operating  proce¬ 
dures.  This  incompatibility  may  make  it  difficult  or  impossible  for  users  to 
relate  their  knowledge  to  the  advice  presented  by  the  aid.  Users  may  be  unable 
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to  provide  the  judgements  needed  by  the  aid  because  they  have  not  thought  about 
the  problem  in  the  terms  used  by  the  aid.  Because  the  aid's  representation  of 
the  problem  does  not  match  the  users,  it  is  perceived  as  incorrect. 

Recommendations:  An  explanation  of  the  aid's  decision  processes  in  terms 

familiar  to  the  users  may  change  their  perception  of  the  aid.  Also  the  aid's 
representation  of  the  problem  and  decision  making  procedures  could  be  cons¬ 
tructed  to  match  the  user's,  if  such  a  representation  does  not  damage  the  aid's 
effectiveness.  If  the  problem  is  that  the  user  does  not  understand  the  aid's 
algorithm,  an  adequate  explanation  of  the  aid  may  be  enough  to  create  accep¬ 
tance.  To  insure  compatibility  between  the  system's  and  users'  representation 
of  the  problem,  users  should  be  incorporated  into  all  phases  of  the  design  and 
development  process.  Where  possible,  follow  representations  of  the  problem  as 
defined  in  field  manuals  and  other  official  publications. 

5 .  Unfamiliar  decision  procedures. 

A  related  problem  is  that  the  aid  procedures  may  seem  unfamiliar  to  the 
user.  They  may  appear  to  be  unnatural,  not  to  fit  the  proolem,  or  not  to  fit 
into  the  larger  decision  making  procedures.  A  variation  of  this  problem  is 
found  when  different  commanders'  decision  making  styles  vary,  and  the  aid  may 
not  support  preferred  procedures.  The  commander  may  have  previously  developed 
a  way  of  thinking  in  his  subordinates  and  now  the  aid  requires  different  proce¬ 
dures  or  a  different  conceptualization  of  the  problem. 

Recommendations:  Recommendations  made  for  the  previous  problem  are  also 

applicable  here.  The  problem  of  different  decision  making  styles  might  be 
addressed  in  an  adaptive  user  interface  that  adapts  the  aid  to  different 
styles.  However,  at  the  present  such  an  adaptive  interface  is  technology 
limited. 


6.  Perception  of  the  aid's  builders. 

Perceived  utility  is  also  affected  by  the  user's  perception  of  the  aid's 
builders.  Government  contractors  may  be  seen  as  not  possessing  the  required 
military  background  to  develop  a  system  with  utility. 

Recommendations:  Education  about  and  demonstration  of  the  aid's  capabili¬ 

ties  will  help  alleviate  this  distrust. 

7 .  Premature  demonstration  of  the  aid. 

A  lack  of  utility  may  be  perceived  if  the  aid  is  demonstrated  during  a 
major  exercise,  and  there  was  not  enough  time  before  the  exercise  to  eliminate 
the  bugs  from  the  system. 

Recommendations:  Any  new  system  will  have  a  break-in  period.  If  organiza¬ 
tional  acceptance  will  be  determined  by  performance  in  field  exercises  then  the 
break-in  period  should  precede  an  exercise.  This  problem  becomes  significant 
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as  we  develop  more  prototype  systems  using  adaptive  design.  The  system  snouia 
be  made  available  to  users  before  the  exercise  so  they  can  become  familiar  with 
it . 

A  lack  of  utility  may  be  perceived  during  the  demonstration  if  users  have 
not  had  enough  training  to  use  the  system  comfortably  and  as  it  was  intended  to 
be  used.  Pretesting  could  determine  how  much  training  is  needed  to  operate  the 
system  comfortably,  and  developers  could  make  sure  users  have  this  training 
prior  to  the  exercises.  Training  embedded  in  the  aid  can  reduce  the  amount  of 
external  training  needed  and  is  also  valuable  as  a  memory  support  if  the  aid  is 
not  used  frequently. 

8 .  The  aid  does  not  speak  "green  suit"  or  "muddy  boot". 

The  language  used,  the  phrasing,  and  the  knowledge  presentation  may  not  be 
those  that  are  ordinarily  used  when  dealing  with  the  problem.  In  addition, 
some  jargon  is  area  or  command  specific.  This  has  implications  for  Army  wide 
systems  or  systems  that  are  to  have  joint  service  applicability. 

Recommendations:  An  aid  should  be  developed  using  a  Subject  Matter  Expert 

(SHE)  from  the  user  world.  A  procedure  which  would  address  this  problem  is 
Adaptive  Design.  Here  SMEs  who  are  interested  in  developing  aids  in  their  area 
of  specialization  are  trained  in  decision  aid  development,  and  in  using  soft¬ 
ware  tools  that  facilitate  aid  development.  (See  the  conclusion  for  a  more 
extended  discussion  of  Adaptive  Design.)  Because  the  SHE  is  developing  the  aid 
he  or  she  will  use  military  language  familiar  to  those  who  will  use  the  aid. 

One  problem  with  this  approach  is  that  the  language  may  be  too  specific  or  too 
colloquial  to  be  used  in  other  functional  areas  or  commands.  However,  using 
military  language  and  representing  knowledge  in  a  way  that  is  familiar  to  the 
user  is  a  key  element  in  developing  user  friendly  and  accepted  systems. 

9.  Lack  of  confidence  in  the  aid's  performance. 

Lack  of  confidence  may  occur  because  the  system  gives  a  "black  box"  solu¬ 
tion  instead  of  a  transparent  one.  The  user  does  not  understand  how  the  system 
functions,  the  basis  for  its  recommendations,  or  the  limitations  of  the  aid, 
and  is  not  able  to  recognize  when  the  system  is  in  error  or  its  use  is  inappro¬ 
priate.  That  is,  the  user  is  responsible  for  decisions  over  which  he  feels  he 
has  no  control  and  does  not  trust  the  system  enough  to  blindly  yield  control. 

Recommendations.  Users  should  have  the  option  of  obtaining  an  explana¬ 
tion  of  how  the  system  works.  The  user  should  be  trained  to  recognize  when 
the  system  is  in  error  and  when  its  use  is  not  appropriate.  However,  the  pro¬ 
vision  of  an  override  capability  is  problematic  because  the  system  may  be 
overriden  inappropriately  if  the  user  prefers  his  own  biased  procedures  to 
those  of  the  aid. 

Evaluation  data  and  results  of  exercises  can  also  help  demonstrate  the 
usefulness  of  the  system  and  increase  confidence  in  the  aid. 
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10.  "The  aid  will  go  away.  Automation  is  a  five  year  fad." 

Unless  the  aid  is  part  of  a  standard  procedure,  it  may  indeed  go  away  when 
its  "champion",  i.e.,  the  person  actively  advocating  the  aid's  use,  moves  on  to 
the  next  assignment.  If  the  aid  wi'l  go  away  shortly,  then  there  is  little 
point  for  users  to  go  through  the  trouble  of  learning  to  use  it  and  of  changing 
their  procedures  to  accommodate  it.  Similarly,  it  may  be  felt  that  automation 
in  general  is  only  a  fad  which  will  go  away  when  it  is  clear  that  it  is  more 
trouble  than  it  is  worth. 

Recommendations.  If  the  user's  perception  that  the  aid  will  go  away  with 
the  champion  is  correct,  then  steps  are  needed  to  ensure  the  institutionaliza¬ 
tion  of  the  aid  and  to  communicate  to  the  user  that  the  organization  can  and 
does  support  the  aid.  The  attitude  that  "automation  is  a  b-year  fad,"  can  be 
changed  by  developinc  and  communicating  to  users  the  long  range  plans  for  auto¬ 
mation.  Users  will  accept  the  aid  if  "the  old  man  accepts  it",  if  it  is  clear 

that  the  aid  or  system  has  the  service's  stamp  of  approval. 

This  problem  is  related  to  a  lack  of  senior  level  commitment  to  the  aid  or 
system.  Commitment  can  be  shown  by  a  statement  of  expectancies,  allocation  of 
staff,  funds  or  training,  or  an  outline  of  the  implementation  plans  and  phas¬ 
ing.  Users  need  to  be  able  to  discern  what  their  supervisors  real  attitudes 
toward  the  proposed  aid  or  system  are.  If  inconsistent  signals  are  being  sent, 
users  will  perceive  a  lack  of  commitment  and  act  accordingly.  Senior  level 

commitment  is  a  big  factor  in  implementation  and  acceptance. 

1 1 .  Inadequate  training. 

Inadequate  training  can  seriously  undermine  the  success  and  acceptance  of 
the  system.  Such  training  can  make  the  system  hard  or  impossible  to  use,  or 
it  can  result  in  partial  or  incorrect  usage  with  the  result  that  the  system 
appears  to  have  less  utility  than  it  actually  has.  In  this  case,  the  system 
may  not  be  used  because  users  do  not  think  it  will  improve  their  performance. 

There  are  several  reasons  why  training  may  be  incomplete.  Developers  are 
not  likely  to  be  training  experts  and  may  not  know  how  much  training  is  needed 
to  optimize  performance  on  the  systems.  Or,  after  training  has  been  completed 
and  the  trainer  has  left,  new  and  unanticipated  difficulties  arise  with  which 
the  user  is  not  equipped  to  deal.  Another  problem  is  that  the  user  organiza¬ 
tion  may  not  want  to  allocate  the  time  and  effort  needed  for  adequate  training. 
The  result  of  such  incomplete  training  is  the  system  will  be  harder  to  use,  may 
not  be  used  fully  or  correctly,  and  consequently  may  show  less  than  its  full 
utility.  All  of  these  can  make  the  users  reluctant  to  use  the  aid. 

Recommendations.  To  insure  that  the  training  is  sufficient  and  at  the 
appropriate  level,  the  vendor's  training  package  should  be  pretested  using 
intended  users.  The  developer  should  also  make  available  training  and  consul¬ 
tations  on  a  continuing  basis,  not  just  a  one  shot  set  of  instruction  classes. 

A  "hoc  line"  to  the  trainer  would  provide  help  with  unexpected  problems  after 
training  is  completed.  A  resident  champion  or  master  user,  who  advocates  and 
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is  knowledgeable  about  the  use  of  the  system,  could  also  provide  the  needed 
on-going  instructional  support  in  addition  to  formal  vendor  training.  A  tech¬ 
nique  with  great  potential  for  improving  the  cost  effectiveness  and  availabil¬ 
ity  of  training  is  embedded  training.  In  embedded  training,  training  and  help 
in  the  use  of  the  aid  is  provided  as  part  of  the  aid  software.  Use  of  embedded 
training  does  not  avoid  the  problem  of  identifying  the  intended  user  and 
designing  effective  instruction  for  him  or  her.  In  fact,  the  importance  of 
these  factors  may  be  magnified  because  the  instructor  is  not  available  to 
answer  unanticipated  questions  or  compensate  for  a  poorly  designed  instruc¬ 
tional  program. 

12.  Personnel  change. 

New  users  are  constantly  coming  in  with  the  rotation  of  Army  personnel. 
These  users  have  to  be  trained,  and  often  training  is  accomplished  by  passing 
it  on  from  user  to  user.  This  method  of  training  is  not  necessarily  bad  but 
could  result  in  the  gradual  degradation  in  the  quality  of  the  training.  Fur¬ 
ther,  user  acceptance  is  then  an  on-going  problem  because  each  new  set  of  users 
must  be  convinced  to  use  the  aid. 

Recommendations .  On-going  vendor  training  and  support,  computer  assisted 
instruction  (CAI)  embedded  in  the  system,  and  very  user  friendly  interfaces 
would  minimize  the  problem.  At  a  broader  level,  it  will  be  necessary  to 
include  Integrated  Logistics  Support  (ILS)  plans  along  with  the  development  of 
decision  aids  themselves.  This  is  especially  important  as  decision  aiding 
moves  from  the  research  environment  to  the  operational  world. 

Another  solution  that  partially  addresses  the  problem  of  rapid  turnover  of 
users  is  to  have  a  standard,  common  interface  within  services  and  across  serv¬ 
ice  systems.  Any  system  to  which  the  user  was  transferred  would  use  the  same 
interface  to  the  aids.  He  could  still  need  training  in  specific  aids  but  much 
of  the  access  procedures,  software  tools,  and  software  sets  would  be  the  same 
no  matter  where  in  the  Army  he  went.  For  example,  there  could  be  a  common  data 
file  format  and  standard  procedures  for  using  the  data  file  no  matter  which 
data  file  was  used. 

13.  Bad  experiences  with  salesmen. 

The  user's  expectations  of  what  the  system  will  do  may  be  too  great  and  he 
then  rejects  the  aid  as  lacking  in  utility  when  actual  performance  falls  short 
of  his  expectations.  In  order  to  obtain  a  contract  or  have  the  system  imple¬ 
mented,  the  contractor  or  developer  may  have  exaggerated  the  aid's  expected 
performance,  minimized  its  limitations,  or  passed  over  its  disadvantages, 
training  requirements,  and  problems.  Part  of  this  problem  may  be  because  the 
builder  may  not  have  an  accurate  picture  of  the  aid  himself.  This  in  turn 
stems  from  inadequate  evaluation  and  testing  of  the  system.  If  the  user  has 
had  bad  experiences  with  salesmen  in  the  past  where  the  capabilities  of  the  aid 
have  been  misrepresented,  the  user  may  be  distrustful  of  the  claimed  capabili¬ 
ties  of  future  aids.  If  he  has  been  burned  once,  he  may  be  unwilling  to  accept 
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future  aids.  On  the  other  hand,  designers  complain  that  aids  do  not  sell  with¬ 
out  "bells  and  whistles".  Aids  that  can  be  implemented  with  current  technology 
are  not  exciting  enough  for  users  who  nay  be  looking  for  an  aid  that  addresses 
important  problems  which  are  obviously  hard  to  solve. 

Part  of  the  gap  between  what  user's  expect  from  the  system  and  the  capabil¬ 
ity  of  the  system  may  stem  from  differences  between  the  user's  and  builder's 
time  frames.  Builders  focus  on  the  future  not  the  present,  and  describe  an 
aid  developed  with  a  technology  ten  years  in  the  future.  The  customer  or  user 

thinks  the  builder  is  describing  an  aid  for  the  here  and  now.  This  problem 

may  be  due  in  part  to  the  acquisition  process  as  it  is  practiced  historically. 
The  length  of  the  acquisition  cycle  guarantees  obsolescence  for  many  aids,  and 

to  circumvent  this  the  designer  may  project  the  technology  he  thinks  will  be 

available  when  the  aid  is  ready  to  be  prototyped.  If  he  guesses  wrong,  the 
needed  technology  will  not  be  there  to  develop  the  aid  as  originally  described. 

Recommendations.  Adaptive  design  would  bypass  the  lengthy  acquisition 
process.  The  user-developer  k’ows  exactly  what  the  aid  can  or  cannot  do  and  do 
not  have  false  expectations  for  the  aid.  Another  recommendation  is  adequate 
evaluation  and  testing  so  that  the  salesman  knows  the  aid's  capabilities, 
likely  problems  and  required  level  of  training.  A  military  champion  responsi¬ 
ble  for  implementation  of  an  aid  could  monitor  contractors'  claims  for  the 
aids.  Adequate  training  should  be  supplied  so  that  full  capabilities  of  the 
aid  are  demonstrated  and  expectations  are  not  unrealized  because  of  lack  of 
training.  All  personnel  -the  developer,  builder,  and  user-  need  to  be  aware 
that  there  will  be  "burn  in"  problems  with  a  new  system  and  that  the  system 
should  not  be  judged  prematurely. 

Damage  to  the  User 

People  see  computers  as  affecting  their  sense  of  self,  jobs,  skills,  poli¬ 
tics,  and  organizational  relationships.  Sometimes  these  perceptions  are  justi¬ 
fied,  sometimes  not.  If  the  perceived  potential  damages  are  too  great,  users 
will  not  accept  or  use  the  system. 

14.  Distrust  of  new  technology. 

Distrust  of  technology  could  be  computer  anxiety,  or  fear  of  the  unknown. 
The  jargon  used  in  the  aid  may  be  different  from  military  jargon  suggesting 
that  the  developer  was  not  military  and  does  not  really  know  the  combat  situa¬ 
tion.  The  new  system  brings  new  equipment  to  learn  to  use  and  service.  The 
user  may  be  afraid  he  would  not  be  able  to  use  the  system  "correctly"  and  will 
appear  unintelligent. 

Recommendations.  This  problem  may  go  away  as  automation  becomes  part  of 
the  normal  operating  procedures  in  the  services.  One  recommendation  is  to  use 
computers  even  more  extensively  in  officer  education  and  training  than  they  are 
presently. 
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15.  Loss  of  control  over  the  decision  making  process. 

The  user  has  ultimate  responsibility  for  decisions.  Because  no  aid  can 
resolve  all  problems  or  be  perfectly  reliable,  the  user  cannot  turn  the  system 
loose  and  blindly  accept  any  and  all  recommendations.  Users  are  then  justifia¬ 
bly  reluctant  to  accept  a  system  over  which  they  have  no  control.  Loss  of 
control  is  also  related  to  another  factor  -  concern  over  decrease  in  personal 
power. 

Recommendations.  Decrease  in  personal  power  and  concern  over  an  ina¬ 
bility  to  control  decision  quality  can  be  addressed  by  building  into  the  aid  an 
override  system  where  the  user  can  substitute  his  own  judgement  for  any  or  all 
parts  of  the  aid  processes.  However,  overrides  may  not  always  be  desirable, 
for  example,  in  cases  where  biases  or  preferred  procedures  lead  the  user  to 
suboptimal  results.  The  aiding  algorithm  should  also  be  transparent  or  expla¬ 
nation  of  the  aid  results  available.  In  cases  where  the  aiding  operations 
cannot  be  easily  explained,  such  as  mathematical  algorithms,  higher  level  or 
more  intuitive  explanations  can  be  supplied.  As  discussed  previously  the  ex¬ 
planations  should  be  in  terms  of  a  language  and  concepts  that  are  compatible 
with  the  user's  usual  way  of  thinking  about  the  problem.  Training,  either 
formal  or  embedded,  should  supply  the  user  with  a  list  of  limitations  of  the 
system  or  situations  where  its  use  is  not  appropriate.  Loss  of  control  over 
decision  quality  can  also  be  addressed  by  creating  trust  in  system  output.  If 
the  user  can  be  shown  by  means  of  experience  with  the  aid,  evaluation  results, 
demonstrations,  or  the  endorsement  of  those  in  authority  that  the  aid  can  be 
trusted,  then  the  user  is  likely  to  trust  and  accept  it. 

16.  Damage  to  skills  and  expertise. 

If  the  decision  maker's  skills  and  expertise  have  been  incorporated  into 
the  aid  and  used  in  place  of  the  decision  maker,  then  over  time  these  skills 
and  expertise  may  erode  if  they  are  no  longer  used.  The  experience  base  which 
was  the  foundation  of  the  DM's  expertise  may  disappear.  This  in  turn  may  lead 
to  excessive  dependence  on  the  aid,  where,  if  it  breaks  down,  adequate  manual 
skills  would  not  exist  to  take  over.  The  seriousness  of  this  problem  depends 
on  the  stage  of  decision  making  that  is  aided.  If  it  is  an  information  aggrega¬ 
tion  stage  or  information  storage  stage,  it  may  not  matter.  Or  if  it  is  a  step 
decision  makers  do  not  do  well  anyway  little  is  lost.  However,  if  the  skills 
Chat  are  aided  are  those  developed  through  long  experience,  loss  of  this  exper¬ 
tise  can  be  serious.  Loss  of  skills  also  depend  on  who  will  use  the  aid.  If 
it  is  intended  for  novices,  few  skills  will  be  lost.  If  it  is  intended  for 
experts,  the  potential  for  loss  of  expertise  is  greater.  As  such,  this  problem 
is  especially  relevant  to  the  development  of  expert  systems. 

Recommendations.  Skill  maintenance  and  practice  using  a  back  up  system  can 
keep  expertise  from  being  lost.  It  is  likely  that  DSS  in  the  near  future  will 
not  automate  all  the  decision  making  steps,  with  a  ready  made  decision  coming 
being  produced.  Rather  they  will  aid  specific  and  limited  steps  within  the 
decision  cycle.  This  may  mean  that  the  decision  making  procedures  will  change 
and  the  nature  of  the  user  expertise  involved  will  also  change.  An  analogous 
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situation  existed  in  the  1950's  when  it  became  possible  to  computerize  statis¬ 
tical  analyses  in  the  social  sciences.  Raw  data  was  fed  into  the  computer  and 
a  series  of  parameters  and  statistics  came  out.  Some  researchers  were  uncom¬ 
fortable  with  this  because  they  liked  to  work  with  the  raw  data  and  to  get  the 
"feel"  of  it.  However,  they  learned  they  didn't  need  to  look  at  the  raw  data 
and  that  they  could  do  their  analyses  faster  and  better  using  only  the  computer 
generated  parameters.  With  the  computer  more  sophisticated  analyses  were  pos¬ 
sible.  Similarly,  with  the  incorporation  of  automated  assistance  into  the 
decision  making  cycle,  although  the  nature  of  decision  making  expertise  may 
change,  the  need  for  human  experts  will  not  vanish.  However,  it  is  important 
that  automated  environments  continue  to  provide  contexts  in  which  such  exper¬ 
tise  can  be  developed.  The  decision  maker  must  not  become  just  a  decision  aid 
operator . 


1 7 .  Damage  to  career  or  status. 

The  user  may  fear  that  with  the  loss  of  the  skills  and  expertise  that  make 
him  valued  and  unique  will  come  also  a  loss  of  status.  Or  he  may  feel  that 
because  the  computer  now  contains  his  expertise,  he  is  not  as  essential  as 
before.  He  may  feel  that  the  computer  will  take  over  his  functions,  there  will 
be  an  erosion  of  his  responsibilities  and  he  will  eventually  be  replaced,  or 
down  graded.  If  he  cannot  exercise  his  skills  and  expertise,  the  user's  job 
satisfaction  may  be  decreased.  He  may  feel  he  will  make  a  wrong  decision 
because  he  does  not  understand  how  the  system  functions,  or  that  he  is  respon¬ 
sible  for  the  quality  of  the  decision  but  does  not  have  control  over  it.  The 
user  may  fear  that  more  will  be  expected  of  him  because  a  tool  is  now  available 
to  do  part  of  his  work. 

Another  perception  of  threat  to  career  may  arise  from  the  prospect  of  hav¬ 
ing  to  learn  new  systems  and  new  skills  and  the  uncertainty  of  how  he  will 
function  in  this  new  environment. 

On  the  other  hand,  the  user  may  think  the  use  of  aids  that  are  not  a  part 
of  the  standard  procedures  could  also  damage  his  career.  Personnel  are 
rewarded  for  staying  within  the  system,  and  use  of  an  aid  not  officially  sanc¬ 
tioned  may  result  in  damage  to  his  career. 

Recommendations.  Part  of  the  fear  of  learning  new  skills  is  a  training 
problem.  The  training  should  be  approached  on  several  levels:  schoolhouse, 
embedded  training,  and  training  using  realistic  simulations  or  an  operational 
setting.  Part  is  also  a  design  problem  in  that  the  system  should  be  as  easy  to 
learn  and  use  as  possible. 

The  problem  of  responsibility  for  decisions  the  user  feels  he  has  no  con¬ 
trol  over  can  be  addressed  by  making  the  system  transparent  so  the  user  knows 
the  origins  of  the  computer  output  and  can  decide  whether  or  not  to  use  this 
output . 
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Another  part  of  the  problem  is  the  proliferation  of  aids  developed  outside 
the  organization.  Mechanisms  are  needed  to  integrate  new  aids  so  that  the  user 
is  not  put  in  a  position  of  using  aids  not  organizationally  sanctioned. 

18.  Increase  in  workload. 

Uorkshop  participants  thought  that  staff  officers  are  overworked  now  and 
use  of  the  aid  may  only  add  more  work.  Use  of  the  system  may  involve  differ¬ 
ent  or  more  cognitive  skills,  and  the  workload  of  the  user  may  be  increased  not 
decreased.  The  computer  may  take  over  much  of  the  routine  work  involved  in 
decision  making  leaving  the  more  difficult  tasks  for  the  user.  Users  may  have 
more  information  to  process,  have  to  learn  about  entirely  new  systems,  or  have 
to  integrate  information  from  multiple  decision  aids.  The  new  system  brings  an 
increased  workload  and  greater  responsibilities.  The  user  may  now  be  responsi¬ 
ble  for  faster  and  higher  quality  decisions.  In  addition,  users  complain  that 
they  need  to  maintain  manual  back  up  systems  in  case  something  happens  to  the 
computer  and  they  must  therefore  maintain  two  systems,  which  doubles  the  amount 
of  work. 


Recommendations.  Objections  to  aids  due  to  workload  are  partly  a  matter  of 
inaccurate  expectations.  It  should  be  emphasized  to  the  user  that  the  savings 
achieved  with  the  aids  is  not  work  but  time  and/or  decision  quality.  Field 
exercises  testing  the  aid  should  also  include  the  practice  of  back  up  systems 
so  that  it  is  clear  that  the  maintenance  of  back  up  systems  is  part  of  the  new 
procedure.  Demonstrations  can  make  clear  just  what  are  the  relative  advantages 
of  the  aid.  Users  should  be  prepared  for  the  change  in  work  composition, 
initial  work  slow  down  and  possibly  added  workload.  The  focus  should  be  on 
increased  effectiveness,  not  on  an  easier  job.  On  the  other  hand,  builders 
should  conduct  workload  analyses  so  that  if  the  workload  has  in  fact  increased, 
the  design  can  be  reconsidered. 

19.  "Real  men  don't  use  keyboards." 

Keyboards  are  traditionally  associated  with  clerks  and  not  with  a  masculine 
combat  environment.  The  officers  for  whom  the  aid  was  designed  may  feel  that 
interacting  with  a  keyboard  is  not  appropriate  for  their  ranks.  Not  all  Uork¬ 
shop  participants  thought  this  factor  was  a  problem  for  user  acceptance  or  even 
that  it  represents  user  attitudes.  Moreover,  this  is  an  attitude  that  may  not 
be  a  problem  in  the  future  as  more  computers  are  put  in  the  field  and  users 
become  more  comfortable  with  them.  Some  officer  schools  furnish  students  with 
computers  which  they  are  expected  to  use  in  doing  their  assignments. 

Recommendations.  Implementation  should  start  at  the  top,  so  that  the 
commander  understands  what  the  aid  can  and  should  do  and  will  convey  his 
expectations  to  the  staff  officer  for  work  at  least  as  good  as  the  aid  will 
support.  Incorporate  the  aiding  system  into  the  school  system  so  that  users 
can  become  comfortable  with  computer  technology  and  with  specific  systems. 


For  those  officers  already  in  the  field,  demonstrations  clearly  showing  the 
advantages  of  using  the  system  will  help  to  override  reluctance  to  use  key¬ 
boards.  Even  more  effective  in  changing  present  officers'  attitudes  may  be  the 
observation  of  the  improved  efficiency  of  new  officers  using  their  computers. 

Aid  is  Hard  to  Use 


An  aid  could  have  a  great  deal  of  utility,  but  if  the  user  cannot  use  it, 
can  use  it  with  difficulty,  or  can  only  use  it  in  a  limited  way,  it  may  not  be 
accepted  by  the  user.  That  is,  the  aid  has  utility  but  the  user  cannot  access 
this  utility.  Items  20  to  22  discuss  reasons  why  an  aid  may  be  hard  to  use. 

20.  Bad  interface  design. 

The  design  of  the  user-computer  interface  could  make  it  difficult  for  the 
user  to  get  the  computer  to  do  what  he  or  she  wants  it  to  do.  For  example  some 
types  of  commands  are  hard  to  remember. 

Recommendations:  Human  factors  considerations  should  be  planned  for  in  the 
design  stage  and  should  be  evaluated  early  enough  so  there  is  still  time  to 
make  substantive  changes  in  the  interface  design.  Human  factor  guidelines  and 
standards  are  available  to  guide  the  interface  design.  However,  guidelines  are 
sometimes  too  general  or  too  specific  for  a  particular  system.  Rapid  proto¬ 
typing  is  a  good  way  to  try  out  and  test  different  interface  configurations. 

21.  Hardware  and  software  incompatibility. 

If  the  aid  is  not  compatible  with  existing  hardware  and  software,  the  aid 
is  not  likely  to.be  used.  There  are  a  number  of  reasons  why  it  might  not  be 
compatible.  Development  and  modification  of  the  aid  might  be  easier  using  one 
set  of  software.  For  example,  an  SHE  developing  an  aid  using  adaptive  design 
may  tend  to  use  an  expert  system  (ES)  tool  with  which  he  or  she  is  familiar  and 

which  he  or  she  considers  easy  to  use.  The  SME  may  feel  he  or  she  will  address 

compatibility  problems  after  the  aid  is  developed.  In  addition,  military  soft¬ 
ware  and  hardware  standards  may  change.  Builders  may  try  to  predict  what 
future  standards  will  be  and  guess  wrong.  Military  "standards"  are  not  consis¬ 
tent  throughout  the  Army,  or  between  services  and  it  is  not  always  clear  which 

set  of  standards  to  use.  With  new  technology  developments,  last  year's  stan¬ 

dards  may  no  longer  be  appropriate. 

Recommendations.  A  layered  approach  to  system  design  should  be  adopted  to 
minimize  the  inevitable  hardware  incompatibilities.  The  user  interface  layer 
should  be  standard,  the  aid  itself  should  be  machine  independent,  and  the 
interaction  between  the  aiding  software  and  the  hardware  should  be  handled  by 
"device  drivers".  These  drivers  are  software  links  between  the  aid  and  exist¬ 
ing  system  hardware  that  can  transform  the  aid  hardware  requirements  into 
requirements  compatible  with  existing  system  hardware. 
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There  are  two  aspects  to  this  integration  problem:  (1)  Aids  are  not 
integrated  into  the  day  to  day  operations  of  the  organization.  At  the  present 
there  is  no  formal  mechanism  for  getting  isolated  aids  developed  and  integrated 
into  the  large  organization's  standard  operating  procedures.  (2)  Aids  are  not 
integratable .  Isolated  aids  are  being  developed  where  each  addresses  part  of  a 
larger  problem,  but  these  aids  don't  fit  together  or  talk  to  each  other.  Each 
aid  executes  a  subtask  of  a  larger  problem,  but  the  aids,  having  been  developed 
by  different  SME's  or  contractors  may  use  a  different  conceptualization  or 
representation  of  the  problem.  One  aid's  output  may  not  be  able  to  feed 
directly  into  the  aid  addressing  the  next  subproblem.  Finally,  if  each  aid  has 
a  different  interface,  the  user  is  faced  with  an  impossible  task  of  learning 
different  interface  conventions  and  switching  between  them. 

Recommendations.  A  common  set  of  functional  or  mil  standards  like  those 
that  define  the  1553  Data  Bus  is  a  preliminary  requirement  for  aids  that  work 
together  or  "talk  to  each  other."  It  is  also  a  primary  prerequisite  for  get¬ 
ting  aids  integrated  into  the  organization.  The  problem  of  getting  isolated 
aids  integrated  is  a  management  problem  that  needs  to  be  addressed. 


INVOLVING  THE  USER  IN  AID  DESIGN  AND  TESTING 


I 

Many  of  Che  recommendations  in  Che  previous  section  suggest  involving  the 
user  in  the  design  and  evaluation  of  the  aid.  Figure  1  shows  the  rationale 
underlying  the  recommendation  to  involve  users  in  aid  design  and  development. 

If  users  are  involved,  designers  can  more  validly  assess  user  requirements  and 
potential  difficulties  with  the  system.  Users  will  have  a  better  understanding 
of  the  system  and  will  be  more  committed  to  the  system  design  if  they  have  I 

helped  to  design  it.  These  three  factors  -  a  better  understanding  of  the  sys¬ 
tem,  a  better  system,  and  commitment  to  the  system  -  lead  to  increased  use  and 
satisfaction  with  the  system.  The  evidence  is  not  clear  whether  increased  use 
causes  increased  satisfaction  or  vice  versa,  but  there  is  probably  an  inter¬ 
active  effect.  In  any  case,  user  involvement  should  lead  to  increased  use. 

I 

However,  user  involvement  is  often  not  easy  to  implement.  This  section 
will  discuss  some  of  the  problems  associated  with  obtaining  users  and  makes 
recommendations  for  addressing  these  problems. 

1.  Potential  military  users  are  busy  people  who  are  overworked  now,  and 
often  engaged  in  projects  from  which  they  cannot  be  spared.  If  they  are  * 

assigned  to  assist  aid  designers  it  may  mean  extra  work  for  them  or  for  their 
colleagues.  Assisting  aid  designers  adds  another  job  to  an  already  impossible 
schedule. 


Recommendations.  Allocation  of  personnel  and  time  for  user  assignment 
to  aid  designers  should  be  built  into  short  and  long  term  military  planning 
cycles.  User  involvement  should  be  part  of  the  activity's  workload.  Top  level 
assignment  would  facilitate  user  involvement.  Use  of  adaptive  design,  where 
the  aid  is  developed  by  the  end  user,  would  also  address  the  problem  of  finding 
users  to  aid  the  design  process. 

Another  recommendation  is  to  co-locate  the  designer/analyst  in  the  user's 
environment.  User's  would  not  have  to  be  pulled  off  of  their  regular  work  and 
the  designer  could  observe  the  users  in  their  usual  work  environments.  This 
method  of  involving  users  has  been  implemented  successfully  in  private  indus¬ 
try,  but  has  the  potential  for  disrupting  user  performance. 

2.  The  section  defining  the  user  discussed  the  point  that  tomorrow's 
complex  decision  support  and  information  systems  will  be  used  by  users  that 
vary  in  functional  area,  service,  experience,  and  a  wide  variety  of  individual 
characteristics.  This  means  that  if  only  one  or  two  users  can  be  provided  to 
assist  aid  design  and  evaluation,  then  it  is  not  clear  which  users  to  select 
and  to  which  users  the  results  can  be  generalized. 

Recommendations.  Draw  users  from  several  environments.  Select  users 
based  on  what  type  of  information  is  needed  for  design  or  evaluation.  Research 
is  needed  to  determine  which  individual  differences  impact  requirements  and 
acceptance . 
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Better  Under¬ 
standing  of 


Figure  1.  Why  involve  users? 


3.  Users  may  be  reluctant  to  participate  in  an  aid  design,  especially  if 
it  is  an  expert  system.  This  may  be  because  the  user  is  protective  of  his  own 
expertise,  because  of  a  skepticism  that  a  micro-chip  could  do  what  he,  the 
expert,  does,  or  because  the  user  fears  that  a  detailed  examination  of  his 
expertise  may  expose  its  flaws. 

Recommendations.  Involve  multiple  SMEs  so  that  the  effect  of  individ¬ 
ual  flaws  will  be  minimized,  and  a  SME  won't  feel  he  has  sole  responsibility 
for  providing  the  expertise  that  will  be  built  into  the  system. 

4.  Ideally,  users  should  be  involved  in  the  design  and  evaluation  of  the 
aid  over  the  life  cycle  of  it  development  and  fielding.  However,  it  is 
unlikely  that  a  user  would  be  assigned  on  a  long  time  basis.  Other  projects 
and  organizations  are  always  competing  for  the  user's  time  and  involvement. 

Even  if  he  were  so  assigned,  the  rotation  system  in  the  services  limits  the 
length  of  involvement ,  and  breaks  the  commitment-interest  continuum  of  the 
user.  Expert  systems  especially  require  a  long  user  commitment  and  management 
may  be  unwilling  to  release  them  for  the  months  or  even  years  the  design  and 
testing  require. 

Recommendations.  The  use  of  rapid  prototyping  can  help  minimize  the 
time  required  for  design  and  evaluation  and  consequently  for  users'  services. 

In  adaptive  design  the  user/designer  takes  the  developing  aid  with  him  to  new 
assignments  and  the  time  required  to  develop  the  aid  is  not  a  problem. 

The  use  of  different  users  over  the  design  and  evaluation  cycle  is  actually 
desirable.  It  ensures  that  the  aid  will  have  utility  for  and  be  usable  by  the. 
whole  class  of  intended  users  and  cancels  any  effects  of  personal  idiosyncra¬ 
sies  of  individuals  involved  in  design  and  evaluation. 

High  level  assignment  of  users  to  design  and  evaluation  duty  will  minimize 
the  importance  of  the  personal  interest-commitment  factor  in  securing  user 
cooperation.  High  level  assignment  would  also  be  necessary  for  obtaining  the 
long  term  services  of  experts  for  ES  development. 


ORGANIZATIONAL  SUPPORT 


Links  Between  the  User,  Builder,  and  Developer 

Figure  2  describes  links  between  the  various  participants  in  aid  design, 
development  and  implementation.  Included  are  those  who  will  actually  use  the 
aid,  i.e.,  the  end  user,  the  decision  maker,  and  the  organizational  usei.  The 
builder  designs,  constructs  and  evaluates  the  aid.  The  user  provides  informa¬ 
tion  to  the  builder  for  requirements  analyses  and  for  test  and  evaluation.  The 
developer  is  responsible  for  aid  development  and  sets  in  motion  and  oversees 
aid  design,  construction  and  evaluation  by  the  builder.  The  developer  obtains 
users  for  the  builder  for  requirements  definition  and  evaluation.  The  devel¬ 
oper  also  provides  opportunities  for  field  testing  the  aid  by  the  builder  and 
deals  with  the  intended  user  organization  to  get  the  aid  incorporated  into  the 
organizational  structure. 

The  change  agent  or  champion  is  an  individual  within  the  organization  who 
enthusiastically  promotes  the  aid,  can  point  out  the  aid's  utility  and  can  help 
in  training  or  trouble  shooting  problems  with  the  aid.  The  champion's  per¬ 
ceived  power  and  authority  are  important  to  whether  the  aid  is  accepted.  The 
change  agent  has  links  to  the  developer,  organization,  and  user. 

Except  for  the  builder-developer  relationship,  the  links  between  the  par¬ 
ticipants  are  usually  informal  and  unstructured.  However,  bridging  the  gaps 
between  the  participants  is  critical  to  both  user  acceptance  and  getting  the 
aid  fielded.  The  user-builder  link  is  especially  problematic.  Anything  that 
can  be  done  to  strengthen  and  formalize  the  links  will  promote  acceptance  of 
the  aid  involved. 

In  adaptive  design,  the  builder,  change  agent  and  user  may  be  the  same 
person.  An  SHE  learns  the  technology  involved  in  developing  an  aid.  He  iden¬ 
tifies  a  task  that  needs  aiding  within  his  area  of  expertise,  and  determines 
the  task  and  knowledge  requirements  to  be  built  into  the  aid.  The  SHE  designs, 
tests,  and  modifies  the  aid  and  uses  and  promotes  the  aid  to  others  and  within 
the  organization.  Because  the  SHE  assumes  many  roles  otherwise  performed  by 
diverse  elements  communication  and  acceptance  are  enhanced.  In  adaptive  design 
a  big  stumbling  block  is  the  weak  link  between  the  SME/builder/user  and  the 
organization.  Generally  there  is  no  formal  link  for  integrating  the  work  of 
the  SHE  into  the  organization.  VJithout  such  a  link  wide  spread  organizational 
acceptance  of  the  aid  is  unlikely.  Adaptive  design  can  address  many  of  the 
factors  that  promote  user  acceptance.  However,  before  it  can  be  truly  feasible 
as  a  design  strategy,  some  mechanism  must  be  created  for  institutionalizing  the 
aid  and  getting  it  in  place  as  part  of  the  organizational  structure. 

A  methodology  that  provides  for  close  communication  between  the  user  and 
builder  is  rapid  prototyping.  A  critical  factor  in  user  acceptance  is  that 
the  system  must  address  a  perceived  need  and  support  improved  performance.  This 
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Linkages  between  participants  in  aid  design,  development,  and  implementation. 
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is  a  requirements  definition  problem.  However,  it  is  not  enough  to  ask  users 
what  they  want.  Often  they  cannot  verbalize  what  they  need  but  will  "know  it 
when  they  see  it",  or  the  system  may  change  task  procedures  so  that  some 
requirements  cannot  be  predicted  in  advance.  Traditionally,  user  requirements 
are  defined  at  the  beginning  of  the  material  development  cycle  and  no  provi¬ 
sions  are  made  for  substantial  modifications  to  those  requirements.  Thus, 
traditional  methods  for  defining  and  validating  user  requirements  are  inade¬ 
quate  for  DSS  development.  Rapid  prototyping  is  a  technique  that  supports  the 
rapid  development  of  successive  versions  of  the  aid,  which  users  can  then  test 
and  evaluate.  Aids  can  then  be  quickly  modified  based  on  user  requirements 
and  tested  again  by  the  user.  This  methodology  provides  close  user-builder 
communication,  the  opportunity  for  the  user  to  test  the  aid  while  his  input  can 
still  substantially  influence  the  design,  and  a  mechanism  for  users  to  react  to 
operable  real  time  simulations.  All  of  these  factors  will  promote  improved 
requirements  specifications  and  user  acceptance  of  the  aid. 

There  are  a  number  of  ways  the  links  between  these  various  groups  can  be 
strengthened: 

•  Develop  formal  organization  structures  to  establish  these  links  and 
facilitate  communications  between  them. 

•  Identify  an  aid  champion,  i.e.  a  single  individual  or  small  group 
who  is  committed  to  the  aid  implementation,  understands  it,  can 
oversee  user  training  and  trouble  shoot  the  break-in  period.  The 
champion  is  a  self  selected  military  individual  who  has  a  personal 
interest  in  the  aid  and  can  interface  with  the  other  groups. 

•  Adaptive  design  is  especially  conducive  to  developing  an  aid  cham¬ 
pion,  but  this  procedure  must  have  organizational  support. 

•  Rapid  prototyping  supports  the  close  interaction  between  user  and 
designer. 

•  Form  small  multi-agency  groups  that  can  operationalize  and  promote 
informal  linkages. 


Organizational  Mechanisms 

With  respect  to  organizational  structure,  mechanisms  are  needed  that  will: 

•  Identify  and  provide  users  for  requirements  specification  and 
evaluation  in  an  on-going  and  systematic  fashion. 

•  Define  requirements. 

•  Be  responsible  for  the  timely  testing  and  aid  modification. 
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RECOMMENDATIONS 


The  recommendations  that  were  made  in  the  preceding  sections  are  listed  in 

this  section. 

User  Involvement : 

Identify  and  involve  the  intended  users  in  aid  design,  development,  and 

evaluation. 

Design  Principles: 

1.  Make  aid's  representation  of  the  problem  match  that  of  the  users. 

2.  Incorporate  an  explanation  capability  in  the  aid. 

3.  Document  and  explain  any  algorithms  and  the  aid’s  logic. 

4.  Consider  the  place  of  the  aid  in  the  existing  task  and  information 
flow.  Aid  should  fit  in  existing  organizational  structure. 

5.  Consider  human  factors  principles  in  the  aid  design.  Test  and  evalu¬ 
ate  for  ease  of  use. 

6.  Incorporate  a  common  interface  across  aids  and  systems. 

Design  and  Implementation  Procedures: 

1.  Use  evolutionary  requirements  analysis  and  design. 

2.  Use  adaptive  design  and  rapid  prototyping  strategies  to  circumvent  many 
of  the  problem  of  an  up-front  requirements  analysis. 

3.  Do  ongoing  test  and  evaluation  in  the  design,  prototype  and  fielding 
stages . 

4.  Provide  break-in  period  for  aid  before  major  field  exercises  which  test 
the  aid. 

5.  Demonstrate  aid's  capabilities  to  users. 

6.  Do  a  workload  analysis  so  aid  does  not  increase  workload. 

7.  Give  adequate  training  in  the  use  of  the  aid  so  that  full  benefit  can 
be  obtained  from  the  aid.  Pretest  the  training  package.  Give  training 
on  the  limitations  and  boundaries  of  the  aid. 

8.  Use  embedded  training  and  built-in  help  facilities. 
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Organizational  Support: 

1.  Include  allocation  of  users  for  test  and  evaluation  as  part  of  large 
and  short  term  planning. 

2.  Seek  public  organizational  commitment  to  the  aid. 

3.  Create  formal  or  informal  organizational  structures  to  establish  links 
between  the  user,  builder,  and  developer  and  to  facilitate  communica¬ 
tion  between  them. 


CONCLUSIONS 


Several  general  recommendations  underlie  most  of  the  recommendations  men- 
tioned  earlier.  These  are:  (1)  involving  the  user,  (2)  sufficient  user  train¬ 
ing,  (3)  evolutionary  requirements  analysis,  and  (A)  careful  organizational 
management.  Each  of  these  is  discussed  below. 

1.  Involve  the  user. 

-  ! 

Involve  the  user  in  aid  design  and  evaluation.  User  involvement  promotes 
commitment  to  the  aid,  aid  utility,  ease  of  use,  a  better  user  understanding  of 
the  system,  and  ensures  the  language  and  problem  representation  will  be  com¬ 
patible  with  the  user's.  All  of  these  are  factors  that  affect  user  acceptance. 
Organizational  support  is  critical  in  identifying  appropriate  users  and  obtain¬ 
ing  cooperation  of  the  users.  ^ 

2 .  Sufficient  user  training. 


Training  was  thought  to  have  a  significant  impact  on  user  acceptance  and 
system  usage  by  affecting  both  ease  of  use  and  perceived  utility.  All  users 
should  receive  training  until  they  are  comfortable  with  the  system.  Training 
could  include  formal  classes,  a  hot  line,  manuals,  embedded  training,  and  on¬ 
going  vendor  support  as  well  as  more  general  training  in  the  use  of  computers. 
Training  implications  should  be  considered  during  design  and  development.  A 
good  design  can  minimize  the  amount  of  training  that  is  needed. 

3 .  Evolutionary  requirements  analysis. 

A  critical  factor  in  user  acceptance  is  that  the  system  must  address  a 
perceived  need  and  support  improved  performance.  This  is  a  requirements  defi¬ 
nition  problem.  However,  it  is  not  enough  to  ask  users  what  they  want.  Often 
they  cani.ot  verbalize  what  they  need  but  will  "know  it  when  they  see  it,"  Or 
the  system  changes  task  procedures  so  that  some  requirements  cannot  be  pre¬ 
dicted  in  advance.  This  means  that  requirements  often  cannot  be  specified  "up 
front"  and  that  the  development  of  aids  does  not  fit  well  into  the  materiel 
acquisition  process  of  the  Services.  In  traditional  acquisition,  requirements 
are  established  up  front,  prior  to  design  and  development.  Test  and  evaluation 
results  only  in  fixes  to  the  existing  system.  However,  often  requirements 
evolve  as  the  aid  is  being  developed.  Two  procedures  that  address  the  require¬ 
ments  definition  problems  are  adaptive  design  and  rapid  prototyping. 

Adaptive  design.  Many  of  the  problems  in  user  acceptance  can  be  addressed 
by  using  a  SME  to  develop  the  aid  in  an  iterative  r  ,shion.  The  SME-developer 
is  then  the  source  of  the  aid  requirements.  This  method  is  being  used  by  the 
Army  Signal  Center,  the  Air  Force  Institute  of  Technology,  and  the  Naval  Post¬ 
graduate  School.  A  key  problem  with  this  method  is  that  there  is  no  formal 
mechanism  to  get  the  aids  integrated  into  the  user  organization,  and  if  this 
method  is  to  be  made  feasible,  this  problem  must  be  addressed  at  the  organiza¬ 
tion  level. 
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Rapid  Prototyping.  This  is  a  technique  which  supports  the  rapid  develop¬ 
ment  of  successive  versions  of  the  aid,  and  which  users  can  then  test  and 
evaluate.  Aids  can  then  be  quickly  modified  based  on  user  requirements  and 
tested  again  by  the  user.  This  iterative  approach  provides  for  close  user- 
designer  communication,  the  opportunity  for  the  user  to  evaluate  the  aid  as  it 
is  developed  when  his  input  can  still  substantially  influence  the  design,  and  a 
mechanism  for  users  to  react  to  operable  real  time  simulations.  All  of  these 
factors  will  promote  improved  requirements  specif ications  and  user  acceptance 
of  the  aid. 

A .  Careful  organizational  management. 

Most  of  the  recommendations  discussed  in  this  report  can  best  be  accomp¬ 
lished  through  careful  organizational  management  of  the  selection,  design,  and 
implementation  of  the  aid  or  DSS.  For  example,  utility  can  be  maximized 
through  systematic  requirements  analysis,  on-going  and  timely  test  and  evalua¬ 
tion,  and  integration  of  the  aid  into  the  larger  organizational  structure.  One 
participant  suggested  that  all  of  the  responsibility  for  user  acceptance 
depends  on  how  the  organization  approaches  the  selection,  design,  and  implemen¬ 
tation  of  an  aid. 

The  organization  should: 

•  Ensure  ongoing  and  reliable  user  availability  for  requirements  analyses, 
and  test  and  evaluation. 

•  Develop  and  employ  a  Life  Cycle  Development  process  that  accommodates  an 
interactive  requirements  analysis,  e.g.,  rapid  prototyping. 

•  Provide  public  commitment  to  the  implementation  of  aids  that  are 
selected. 


•  Develop  mechanisms  which  will  provide  for  formal  links  between  the  user, 
developer,  and  builder. 

The  problem  of  user  acceptance  is  a  complicated  one,  not  ultimately  solved 
with  quick  and  easy  cosmetic  fixes.  The  solutions  lie  in  aids  that  both  actu¬ 
ally  and  apparently  respond  to  real  needs  of  the  users  and  in  an  organizational 
structure  that  can  facilitate  and  formalize  the  links  between  the  user,  combat 
developer,  and  builder. 
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